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The  Appropriateness  of  Using  a  Medical  Procedure 

Is  Information  in  the  Medical  Record  Valid? 

Jacqueline  Kosecoff,  PhD,*T  Arlene  Fink,  PhD,*  f 
Robert  H.  Brook,  MD.  ScD,*  $  and  Mark  R.  Chassin,  MD:f 


Understanding  the  clinical  appropriateness  of  a  procedure's  use  may  be  critical 
in  explaining  geographic  variations  in  its  use.  Little  is  known,  however,  about 
whether  data  on  appropriateness  can  be  obtained  from  a  medical  record.  A 
national  panel  of  physicians  formulated  a  list  of  300  mutually  exclusive,  detailed 
clinical  indications  for  performing  coronary  angiography.  Using  this  list,  we 
compared  the  reasons  physicians  perform  coronary  angiography  as  revealed  in 
medical  records  with  those  given  in  interviews  with  the  physicians  who  actually 
did  the  procedure.  Thirty-five  of  47  eligible  billing  entities  (74%)  from  two  Los 
Angeles  Professional  Standards  Review  Organization  areas  participated.  These 
physicians  practiced  in  14  hospitals  and  accounted  for  81%  of  all  angiographies 
performed  on  Medicare  patients  in  the  two  areas.  Sixty-six  records  (approxi¬ 
mately  two  per  physician)  were  reviewed;  physician  interviews  were  conducted 
by  two  trained  data  collectors  who  were  blinded  to  each  other's  results.  Ninety- 
one  percent  agreement  was  reached  on  the  specific  indication  for  performing 
coronary  angiography  when  information  from  the  record  review  and  interview 
was  compared.  We  conclude  that  medical  records  yield  valid  information  on 
why  coronary  angiography  is  performed  and  that  they  are  a  suitable  source  to 
use  in  judging  the  appropriateness  of  that  use.  Key  words:  use  of  medical  records; 
validity  of  medical  record  information.  (Med  Care  1987;  25:196-201) 
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have  been  shown  to  vary  dramatically  by 
geographic  area.1'3  We  do  not  know'  whether 
rates  in  high-use  localities  reflect  too  much 
care  or  those  in  low-use  areas  too  little  care. 
The  variations  have  been  attributed  to  dif¬ 
ferences  in  the  availability  of  financial  and 
technical  resources,  the  needs  and  prefer¬ 
ences  of  patients,  and  uncertainty  about  how 
to  approach  specific  medical  problems.4'6  To 
assess  the  appropriateness  of  the  use  of  a 
specific  procedure,  however,  we  must  be  able 
to  identify  at  a  detailed  clinical  level  why  it 
was  performed.  Two  potential  sources  of  in¬ 
formation  are  medical  records  and  physician 
interviews. 

Acceptance  of  the  medical  record  as  a 
valid  source  of  information  to  judge  the  ap- 
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propriateness  of  care  is  by  no  means  uni¬ 
versal.  The  medical  record  may  not  be  com¬ 
plete,  even  though  it  is  an  essential  source 
of  information  on  the  delivery  of  care.7'" 
Then  why  rely  on  the  medical  record?  Why 
not  just  ask  physicians  their  reasons  for  per¬ 
forming  a  service  and  then  use  that  infor¬ 
mation  to  determine  appropriateness?  Inter¬ 
views  consume  physicians'  time,  disrupt  of¬ 
fice  routines,  and  are  impractical  to  perform 
(the  interviewer  and  physician  must  be 
available  at  a  mutually  convenient  time). 

This  study  used  information  from  medical 
records  and  physician  interviews  to  compare 
the  reasons  physicians  perform  coronary 
angiography.  We  selected  coronary  angiog¬ 
raphy  because  it  is  performed  relatively  fre¬ 
quently  and  its  use  varies  widely  by  geo¬ 
graphic  region,  in  the  United  States,  among 
people  65  years  of  age  or  older,  for  example, 
the  age-sex-adjusted  rate  of  the  use  of  cor¬ 
onary  angiography  varies  at  least  200% 
among  areas  as  large  as  individual  states.1 

Methods 

Sample 

The  Professional  Standards  Review  Or¬ 
ganization  (PSRO)  program  divided  Los  An¬ 
geles  County  into  seven  geographic  areas. 
For  convenience,  we  used  two  of  the  seven 
areas  to  define  our  sample.  We  chose  them 
because  they  were  geographically  distant 
from  one  another  and  contained  patients  of 
varying  socioeconomic  status.  These  pa¬ 
tients,  when  hospitalized,  received  care 
mostly  at  nonteaching  hospitals.  To  avoid 
biases  in  the  quality  of  medical  records  per¬ 
haps  associated  with  a  hospital's  teaching 
status,  we  excluded  the  PSRO  areas  contain¬ 
ing  UCLA  and  the  University  of  Southern 
California. 

To  obtain  our  patient  sample,  we  used 
data  from  the  Part  B  Medicare  carrier  (the 
agency  that  reimburses  physician  services 
under  Medicare)  for  Los  Angeles  County. 
Fifty-six  physicians  or  groups  of  physicians 
(billing  entities)  in  the  two  PSRO  areas  had 


been  reimbursed  by  Medicare,  between  Au¬ 
gust  1982  and  April  1983  for  performing  at 
least  five  coronary  angiographies.  They  ac¬ 
counted  for  97%  of  all  coronary  angiogra¬ 
phies  performed  on  people  65  years  of  age 
or  older. 

Of  the  56  billing  entities,  five  were  ex¬ 
cluded  because  their  last  angiography  was 
performed  in  1982;  three  entities  were  ex¬ 
cluded  because  their  offices  were  geograph¬ 
ically  inconvenient  to  reach;  and  one  was 
excluded  because  it  was  a  teaching  hospital 
affiliated  with  UCLA.  The  remaining  47  bill¬ 
ing  entities  accounted  for  90%  of  the  coro¬ 
nary  angiographies. 

After  phone  calls  or  contacts,  35  (74%)  of 
the  billing  entities  agreed  to  participate  in 
the  study.  We  did  not  collect  formal  infor¬ 
mation  on  reasons  for  not  participating.  The 
most  commonly  cited  reason  was  unwilling¬ 
ness  to  allow  researchers  to  review  medical 
records,  which  may  or  may  not  be  related  to 
the  quality  of  record-keeping.  The  partici¬ 
pating  billing  entities  included  54  (84%)  of 
the  individual  physicians  performing  coro¬ 
nary  angiography  in  the  two  PSRO  catch¬ 
ment  areas  and  accounted  for  81%  of  all  an¬ 
giographies.  The  54  physicians  performed 
angiography  at  14  hospitals,  all  of  which 
agreed  to  participate  in  the  study. 

When  a  billing  entity  was  synonymous 
with  an  individual  physician,  the  medical 
records  of  the  physician’s  three  most  recent 
coronarv  angiography  patients  were  selected 
for  inclusion  in  the  sample.  For  billing  en¬ 
tities  that  represented  a  group  of  physicians 
who  used  a  single  billing  number,  the  three 
most  recent  angiographies  were  selected 
such  that  the  records  of  no  more  than  two 
physicians’  patients  were  reviewed. 

Indications  for  Performing 
Coronary  Angiography 

A  comprehensive  and  mutually  exclusive 
list  of  300  clinically  detailed  indications  or 
reasons  for  performing  coronary  angiogra¬ 
phy  was  prepared.  The  list  was  based  on  the 
findings  of  a  literature  review  and  the  advice 
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Table  1.  Selected  Indications  for  Coronary  Angiography 

1.  Asymptomatic  Patients 

Coronary  angiography  is  indicated  in  patients  in  high-risk  occupations  if  there  is  a  positive  exercise  ECG  and 
no  exercise  thallium  scan. 

2.  Chest  Pain  of  Uncertain  Origin 

Coronary  angiography  is  indicated  in  patients  with  negative  exercise  ECG  and  no  or  negative  exercise 
thallium  scan 

3.  Chronic  Stable  Angina 

Coronary  angiography  is  indicated  in  patients  (without  strong  contraindications  to  coronary  artery  bypass 
graft  surgery)  in  whom  angina  occurs  with  mild  exertion  (Class  III  or  IV)  and  who  have  received  no  or  less 
than  maximal  medical  management  and  no  exercise  ECG,  no  exercise  thallium  scan,  and  no  exercise  MUGA 

4.  Unstable  Angina 

Coronary  angiography  is  indicated  in  patients  (without  strong  contraindications  to  coronary  artery  bypass 
graft  surgery)  during  the  hospital  admission  for  unstable  angina  only  if  pain  persists  after  admission  despite 
maximum  medical  inpatient  management. 

5.  During  an  Acute  Myocardial  Infarction  (MI) 

Coronary  angiography  is  indicated  in  the  acute  phase  of  an  Ml  in  patients  without  strong  contraindications 
to  coronary  artery  bypass  graft  surgery  if  the  MI  is  complicated  by  persistent  chest  pain. 

6.  Within  Six  Months  of  an  acute  Myocardial  Infarction  (Ml) 

Coronary  angiography  is  indicated  in  patients  within  six  months  of  an  Ml  (without  strong  contraindications 
to  coronary  artery  bypass  graft  surgery),  if  the  Ml  was  subendocardial  and  angina  occurs  following  the  MI 
during  mild  exertion  and  the  patient  received  no  or  less  than  maximal  medical  management  regardless  of 
exercise  ECG  results. 

7.  Sudden  Death  Cardiac  Survivors 

Coronary  angiography  is  indicated  in  patients  (without  strong  contraindications  to  coronary  artery  bypass 
graft  surgery)  if  the  sudden  death  episode  was  not  associated  with  an  MI  and  angina  occurs  following  the 
episode  of  sudden  death  during  mild  exertion  (Class  III  or  IV),  and  patient  has  received  no  or  less  than 
maximal  medical  management,  regardless  of  exercise  ECG  results 

8.  Following  Coronary  Artery  Bypass  Graft  Surgery  (CABS) 

Coronary  angiography  is  indicated  in  patients  with  angina  after  CABS  that  occurs  wi.h  mild  exertion  (Class 
III  or  IV),  and  who  have  received  no  or  less  than  maximal  medical  management  regardless  of  exercise  ECG 
results 

9.  Other 

Coronary  angiography  is  indicated  in  the  preoperative  evaluation  of  patients  with  cardiac  valvular  disease. 
Source:  Chassin.'2 


of  a  national  panel  of  physicians.12 13  Indi¬ 
cations  were  grouped  by  clinical  presenta¬ 
tion:  asymptomatic  (N  =  28);  chest  pain  of 
uncertain  origin  (N  =  30);  chronic  stable  an¬ 
gina  (N  =  108);  unstable  angina  (N  =  28); 
acute  myocardial  infarction  (N  =  5);  within 
6  months  of  an  acute  myocardial  infarction 
(N  =  68);  sudden  death  survivors  (N  =  6); 
following  coronary  artery  bypass  graft  sur¬ 
gery  (N  =  16);  and  others  (N  =  1 1).  Numbers 
in  parentheses  indicate  the  number  of  spe¬ 
cific  indications  in  each  clinical  grouping. 

As  illustrated  in  Table  1,  each  indication 
represents  a  specific  set  of  clinical  circum¬ 
stances  under  which  it  might  or  might  not 
be  appropriate  to  perform  coronary  angiog¬ 
raphy.  The  national  panel  that  assisted  in 
devising  this  catalog  of  indications  also  rated 


each  indication  on  a  nine-point  scale  of  ap¬ 
propriateness.12  Thus,  if  we  are  able  to  obtain 
clinical  data  that  are  sufficiently  detailed  to 
classify  a  particular  case  into  one  of  the  300 
indications,  we  have  sufficient  information 
with  which  to  judge  appropriateness. 

Data  Collection 

Two  data  collectors,  neither  of  whom  was 
a  physician,  were  trained  to  review  medical 
records  and  to  interview  physicians.  Each 
record  review  required  about  1  hour  and 
each  interview  about  15  minutes.  The  po¬ 
tential  sample  was  105  patients  (35  billing 
entities  times  three  records  per  entity).  Of 
the  105,  70  (two  from  each  billing  entity) 
were  to  be  used  for  comparing  medical  rec- 
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ord  reviews  with  interviews,  and  35  were  to 
serve  as  reliability  checks.  The  later  records 
were  abstracted  by  both  data  collectors,  but 
neither  one  knew  which  records  were  ab¬ 
stracted  twice. 

When  comparisons  between  interviews 
and  office  records  were  to  be  made,  one  data 
collector  abstracted  the  office  record  of  the 
physician  who  performed  the  angiography 
and  its  corresponding  hospital  record,  while 
the  second  interviewed  the  physician.  In 
three  instances,  because  the  cardiologist  who 
performed  the  procedure  did  not  actually 
evaluate  the  patient,  we  interviewed  and 
abstracted  the  record  of  the  cardiologist  who 
did  evaluate  the  patient  prior  to  the  proce¬ 
dure.  In  each  of  these  three  cases,  both  car¬ 
diologists  were  members  of  the  same  group. 
The  data  collectors  did  not  discuss  their 
findings  with  each  other  and  were  randomly 
assigned  to  either  a  record  abstraction  cr  in¬ 
terview. 

To  standardize  the  medical  records  review, 
we  prepared  detailed  medical  record  ab¬ 
straction  forms  and  guidelines  that  asked  for 
108  specific  bits  of  data  that,  taken  together, 
would  reveal  the  indications  for  the  patient's 
coronary  angiography  .12  To  guide  the  inter¬ 
view  process,  a  two-part  form  was  prepared, 
with  which  we  asked  physicians  to  specify 
their  reasons  for  performing  coronary  an¬ 
giography.  During  the  interview,  the  phy¬ 
sician  had  access  to  all  information  included 
in  the  patient's  medical  record.  The  first  part 
of  the  interview  was  open-ended  and  con¬ 
sisted  of  two  questions.  1)  "Can  you  give  the 
main  or  primary  reasons  why  you  performed 
angiography  on  (give  name  of  patient)  on 
(give  date  of  marker  procedure)?  Please  be 
as  specific  as  possible."  2)  "Are  there  any 
other  factors,  like  test  results  or  functional 
status  data,  that  affected  your  decision? 
Again,  please  be  as  specific  as  possible."  In 
both  cases,  the  physician's  exact  response 
was  copied.  The  second  part  of  the  interview 
was  highly  structured  (Fig.  1).  With  the 
medical  record  still  available,  physicians 
were  asked  to  pick  one  or  more  of  300  mu¬ 


tually  exclusive  indications  for  which  the 
coronary  angiography  had  been  performed. 

Complete  medical  record  and  interview 
data  were  available  for  66  of  the  70  patients 
(94%).  The  four  patients  eliminated  had  their 
angiography  performed  in  a  hospital  that 
was  out  of  our  geographic  area. 

Data  on  reliability  was  obtained  from  34 
of  the  35  eligible  cases  (97%).  As  above,  the 
one  missing  case  represents  a  procedure  per¬ 
formed  in  an  out-of-area  hospital. 

Results 

Relationships  Between  Medical  Records 
and  Interviews 

For  60  of  the  66  records  (9 1  %),  the  medical 
record  review  and  the  interview  agreed  on 
the  indications  for  performing  the  coronary 
angiography.  For  two  of  the  six  patients  in 
which  the  methods  produced  disagreement, 
information  was  provided  in  the  interview 
that  was  not  asked  for  in  our  medical  record 
abstract  form.  This  problem  was  subse¬ 
quently  rectified;  that  is,  the  abstract  form 
was  changed  to  include  questions  designed 
to  obtain  this  information.  For  two  other  pa¬ 
tients,  information  provided  in  the  interview 
conflicted  with  data  in  the  medical  record, 
and  agreement  could  not  be  reached  on 
which  source  was  conect.  Fot  the  last  two 
patients,  the  interview  provided  information 
that  was  contradicted  by  objective  data  in 
the  record;  for  these  patients,  we  believe  the 
medical  records  to  be  correct. 

Agreement  Between  Data  Collectors 

Our  criterion  for  establishing  the  reliability 
of  the  medical  record  abstraction  process  was 
agreement  between  the  abstractors  on  the 
indication  for  performing  the  angiography. 
For  33  out  of  34  cases  (97%),  this  criterion 
was  met.  In  addition,  when  specific  items 
from  the  abstraction  form  were  scrutinized, 
there  was  80%  agreement.  This  level  of  re¬ 
liability,  however,  was  improved  when  all 
different  items  measuring  the  same  concept 
were  grouped.  Data  collectors  sometimes 
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2.  Medical  Managamant  of  Angina  ^ 


Fig.  1 .  A  portion  of  a  structured  physician  interview  used  to  identify  the  reason  for  which  the  coronary  angiography 
was  performed. 
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missed  a  fact  when  it  was  asked  in  one  fash¬ 
ion  but  usually  found  it  when  asked  in  an¬ 
other.  For  example,  under  a  question  about 
arrhythmias,  a  data  collector  missed  the 
presence  of  multifocal  premature  ventricular 
contractions  but  later  noted  this  finding  as 
part  of  an  ECG  interpretation. 

Discussion 

The  purpose  of  our  study  was  to  determine 
the  validity  of  medical  records  in  assessing 
the  reasons  for  performing  coronary  angiog¬ 
raphy.  If  the  results  from  a  record  review 
concurred  with  those  from  physician  inter¬ 
views,  then  collecting  clinical  data  on  why 
such  procedures  are  performed  would  not 
be  dependent  on  interviewing  physicians. 

In  attempting  to  identify  one  or  more  in¬ 
dications  from  a  list  of  300,  we  found  91% 
agreement  between  data  obtained  from  a 
medical  record  and  the  physician's  stated 
reason  for  performing  coronary  angiography. 
When  disagreement  occurred,  it  was  not  ev¬ 
ident  which  data  source  was  correct.  It 
should  be  noted  that  this  level  of  agreement 
was  obtained  from  a  community-based 
sample  of  physicians  performing  angiogra¬ 
phy  in  the  Los  Angeles  area.  We  have  no 
reason  to  believe  that  their  medical  records 
were  any  more  complete  than  those  of  phy¬ 
sicians  in  other  areas  of  the  country.  Thus, 
these  findings  may  be  generalizable,  al¬ 
though  this  latter  assertion  should  be  tested. 
We  also  found  the  abstraction  process  to  be 
reliable. 

Obtaining  these  results  was  dependent  on 
having  well-trained  abstractors  spend  at  least 
1  hour  abstracting  each  medical  record  and 
on  the  use  ot  a  detailed  abstraction  form  and 
accompanying  guidelines.  The  preparation 
of  the  abstraction  system  took  about  1  year 


and  required  the  expertise  of  a  multidisci¬ 
plinary  team  of  researchers  and  physicians 
and  a  national  panel  of  expert  clinicians. 

We  conclude  that  medical  records  can  be 
reliable  and  valid  sources  of  information  to 
identify  the  specific  clinical  reasons  that 
physicians  perform  coronary  angiography. 
How  generalizable  these  findings  are  to  other 
procedures  should  be  examined. 
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